The present application relates to an image processing apparatus and method by which a motion vector between two screen images is detected and the reliability of the detected motion vector is calculated.
A block matching technique which determines a motion vector between two screen images from image information itself is a technique of an old history.
According to the block matching technique, a motion vector between two screen images including a reference screen image which is a noticed screen image and an original image (hereinafter referred to as target screen image) from which a motion of the reference screen image originates is calculated by calculating a correlation between the reference screen image and the target screen image with regard to blocks of a rectangular region of a predetermined size (a target block and a reference block). The block matching technique includes two cases including a case wherein the target screen image precedes in time to the reference screen image as in the case of, for example, motion detection by the MPEG (Moving Picture Experts Group) system, and another case wherein the reference screen image precedes in time to the target screen image as in the case of, for example, noise reduction by superposition of image frames hereinafter described.
It is to be noted that, in the present specification, the term screen image signifies an image formed from image data of one frame or one field. However, for the convenience of description, in the following description of the present specification, it is assumed that one screen image is formed from one frame and is hereinafter referred to as frame. Accordingly, a reference screen image is hereinafter referred to as reference frame and a target screen image is hereinafter referred to as target frame.
A motion vector matching technique by the block matching has been developed particularly in regard to pan-tilt detection and image pickup object tracking of a television camera, moving picture coding of the MPEG (Moving Picture Experts Group) system and so forth. In the nineties, application over a wide range including sensorless camera shake correction by superposition of images, noise reduction upon image pickup in low illuminance and so forth has been promoted.
Further, the motion vector detection method by the block matching is applied not only for the image recognition applications and the camera shake correction applications but also for new applications like automatic adjustment of the shutter speed of an image pickup apparatus and double-speed frame rate conversion of a liquid crystal television set.
In any case, the reliability of a detected vector is significant. In particular, if the detected motion vector is accurate, then positioning of a plurality of images in a high degree of accuracy is possible and tracking of an image pickup object, highly accurate camera shake correction and so forth can be anticipated. However, if a motion vector of low reliability is used to execute a process, then correction or the like is carried out in an erroneous direction, and therefore, a situation that also the accuracy of an application-dependent post-processing step becomes low occurs.
Therefore, a method of extracting a motion vector of high reliability has been proposed and is disclosed, for example, in Japanese Patent Laid-Open No. 2006-318345. According to the method, comparison of AC components and a DC component of a result of discrete cosine transform (DCT) in encoding of the MPEG system is carried out or comparison with an edge of an image is carried out to detect a motion vector from a characteristic image portion as a motion vector having high reliability.
Also a method is available wherein, before vector calculation, characteristic points or singular points in an image are extracted in advance by edge detection or by a section of a corner detector, and a motion vector is determined from the characteristic points.
Further, it is disclosed in Japanese Patent Laid-Open No. 2005-301984 that an index to reliability of a motion vector is obtained from an activity representative of a characteristic amount of an image.